Critical reading of systematic review articles.

A systematic review provides an overview of primary studies investigating a given research question, e.g., the effect of a certain treatment. Individual study results are sometimes synthesised in a meta-analysis. A critical reader should consider whether the systematic review is relevant and reliable, e.g., does it follow a protocol, address the risk of bias, and consider potential heterogeneity. PRISMA 2020 guideline recommends a minimum set of items that should be reported in a systematic review article, and AMSTAR 2 and ROBIS are tools for critical appraisal of systematic reviews.

A scoping review on the methodological and reporting quality of scoping reviews in China.

BACKGROUND: Scoping reviews have emerged as a valuable method for synthesizing emerging evidence, providing a comprehensive contextual overview, and influencing policy and practice developments. The objective of this study is to provide an overview of scoping reviews conducted in Chinese academic institutions over the last decades. METHOD: We conducted a comprehensive search of nine databases and six grey literature databases for scoping reviews conducted in Chinese academic institutions. The reporting quality of the included reviews was assessed using the Preferred Reporting Items for PRISMA-ScR checklist. We performed both quantitative and qualitative analyses, examining the conduct of the scoping reviews and exploring the breadth of research topics covered. We used Chi-squared and Wilcoxon rank-sum tests to compare methodological issues and reporting quality in English and Chinese-language reviews. RESULTS: A total of 392 reviews published between 2013 and 2022 were included, 238 English-reported reviews and 154 Chinese-reported reviews, respectively. The primary purposes of these reviews were to map and summarize the evidence, with a particular focus on health and nursing topics. 98.7% of reviews explicitly used the term "scoping review", and the Arksey and O'Malley framework was the most frequently cited framework. Thirty-five English-reported scoping reviews provided a protocol for scoping review. PubMed was the most common source in English-reported reviews and CNKI in Chinese-reported reviews. Reviews published in English were more likely to search the grey literature (P = 0.005), consult information specialists (P < 0.001) and conduct an updated search (P = 0.012) than those in Chinese. Reviews published in English had a significantly high score compared to those published in Chinese (16 vs. 14; P < 0.001). The reporting rates in English-reported reviews were higher than those in Chinese reviews for seven items, but lower for structured summary (P < 0.001), eligibility criteria (P < 0.001), data charting process (P = 0.009) and data items (P = 0.015). CONCLUSION: There has been a significant increase in the number of scoping reviews conducted in Chinese academic institutions each year since 2020. While the research topics covered are diverse, the overall reporting quality of these reviews is need to be improved. And there is a need for greater standardization in the conduct of scoping reviews in Chinese academic institutions.

Investigate the relationship between the retraction reasons and the quality of methodology in non-Cochrane retracted systematic reviews: a systematic review.

BACKGROUND: This systematic review aimed to investigate the relationship between retraction status and the methodology quality in the retracted non-Cochrane systematic review. METHOD: PubMed, Web of Science, and Scopus databases were searched with keywords including systematic review, meta-analysis, and retraction or retracted as a type of publication until September 2023. There were no time or language restrictions. Non-Cochrane medical systematic review studies that were retracted were included in the present study. The data related to the retraction status of the articles were extracted from the retraction notice and Retraction Watch, and the quality of the methodology was evaluated with the AMSTAR-2 checklist by two independent researchers. Data were analyzed in the Excel 2019 and SPSS 21 software. RESULT: Of the 282 systematic reviews, the corresponding authors of 208 (73.75%) articles were from China. The average interval between publish and retraction of the article was about 23 months and about half of the non-Cochrane systematic reviews were retracted in the last 4 years. The most common reasons for retractions were fake peer reviews and unreliable data, respectively. Editors and publishers were the most retractors or requestors for retractions. More than 86% of the retracted non-Cochrane SRs were published in journals with an impact factor above two and had a critically low quality. Items 7, 9, and 13 among the critical items of the AMSTAR-2 checklist received the lowest scores. DISCUSSION AND CONCLUSION: There was a significant relationship between the reasons of retraction and the quality of the methodology (P-value < 0.05). Plagiarism software and using the Cope guidelines may decrease the time of retraction. In some countries, strict rules for promoting researchers increase the risk of misconduct. To avoid scientific errors and improve the quality of systematic reviews/meta-analyses (SRs/MAs), it is better to create protocol registration and retraction guidelines in each journal for SRs/MAs.

Guidance to best tools and practices for systematic reviews.

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy. A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work. Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Effect of PRISMA 2009 on reporting quality in systematic reviews and meta-analyses in high-impact dental medicine journals between 1993-2018.

INTRODUCTION: The PRISMA guidelines were published in 2009 to address inadequate reporting of key methodological details in systematic reviews and meta-analyses (SRs/MAs). This study sought to assess the impact of PRISMA on the quality of reporting in the full text of dental medicine journals. METHODS: This study assessed the impact of PRISMA (2009) on thirteen methodological details in SRs/MAs published in the highest-impact dental medicine journals between 1993-2009 (n = 211) and 2012-2018 (n = 618). The study further examined the rate of described use of PRISMA in the abstract or full text of included studies published post- PRISMA and the impact of described use of PRISMA on level of reporting. This study also examined potential effects of inclusion of PRISMA in Instructions for Authors, along with study team characteristics. RESULTS: The number of items reported in SRs/MAs increased following the publication of PRISMA (pre-PRISMA: M = 7.83, SD = 3.267; post-PRISMA: M = 10.55, SD = 1.4). Post-PRISMA, authors rarely mention PRISMA in abstracts (8.9%) and describe the use of PRISMA in the full text in 59.87% of SRs/MAs. The described use of PRISMA within the full text indicates that its intent (guidance for reporting) is not well understood, with over a third of SRs/MAs (35.6%) describing PRISMA as guiding the conduct of the review. However, any described use of PRISMA was associated with improved reporting. Among author team characteristics examined, only author team size had a positive relationship with improved reporting. CONCLUSION: Following the 2009 publication of PRISMA, the level of reporting of key methodological details improved for systematic reviews/meta-analyses published in the highest-impact dental medicine journals. The positive relationship between reference to PRISMA in the full text and level of reporting provides further evidence of the impact of PRISMA on improving transparent reporting in dental medicine SRs/MAs.

Guidance to Best Tools and Practices for Systematic Reviews.

¼ Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.¼ A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.¼ Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Efficacy and safety of acupuncture in the treatment of foot drop in post-stroke: A protocol for systematic review and meta-analysis.

BACKGROUND: Foot drop is a common complication in post-stroke. Patients with foot drop are at high risk for falls and fall-related injuries. Accordingly, it can reduce independence and quality of life in patients. Clinical studies have confirmed that acupuncture is effective in treating foot drop in post-stroke. However, there is a lack of systematic review exploring the efficacy and safety of acupuncture treatment. This study aims to assess the efficacy and safety of acupuncture in the treatment of foot drop in poststroke from the results of randomized controlled trials. METHODS: We will search articles in 8 electronic databases including the Cochrane Central Register of Controlled Trials, the Web of Science, PubMed, Embase, the China National Knowledge Infrastructure, the Chinese Biomedical Literature Database, Wanfang Data Database, and the Chinese Scientific Journal Database for RCTs of acupuncture treated foot drop in post-stroke from their inception to 10 August 2022. We will analyze the data meeting the inclusion criteria with the RevMan V.5.4 software. Two authors will assess the quality of the study with the Cochrane collaborative risk bias tool. We will evaluate the certainty of the estimated evidence with the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) method. Data analysis will be performed using STATA 16.0. RESULTS: This study will review and evaluate the available evidence for the treatment of foot drop in post-stroke using acupuncture. CONCLUSION SUBSECTIONS: This study will determine the efficacy and safety of acupuncture applied to post-stroke individuals with foot drop.

The Campbell Collaboration's systematic review of school-based anti-bullying interventions does not meet mandatory methodological standards.

BACKGROUND: Many published reviews do not meet the widely accepted PRISMA standards for systematic reviews and meta-analysis. Campbell Collaboration and Cochrane reviews are expected to meet even more rigorous standards, but their adherence to these standards is uneven. For example, a newly updated Campbell systematic review of school-based anti-bullying interventions does not appear to meet many of the Campbell Collaboration's mandatory methodological standards. ISSUES: In this commentary, we document methodological problems in the Campbell Collaboration's new school-based anti-bullying interventions review, including (1) unexplained deviations from the protocol; (2) inadequate documentation of search strategies; (3) inconsistent reports on the number of included studies; (4) undocumented risk of bias ratings; (5) assessments of selective outcome reporting bias that are not transparent, not replicable, and appear to systematically underestimate risk of bias; (6) unreliable assessments of risk of publication bias; (7) use of a composite scale that conflates distinct risks of bias; and (8) failure to consider issues related to the strength of the evidence and risks of bias in interpreting results and drawing conclusions. Readers who are unaware of these problems may place more confidence in this review than is warranted. Campbell Collaboration editors declined to publish our comments and declined to issue a public statement of concern about this review. CONCLUSIONS: Systematic reviews are expected to use transparent methods and follow relevant methodological standards. Readers should be concerned when these expectations are not met, because transparency and rigor enhance the trustworthiness of results and conclusions. In the tradition of Donald T. Campbell, there is need for more public debate about the methods and conclusions of systematic reviews, and greater clarity regarding applications of (and adherence to) published standards for systematic reviews.

Practical guide to undertaking scoping reviews for pharmacy clinicians, researchers and policymakers.

WHAT IS KNOWN AND OBJECTIVE: Scoping reviews are a valuable evidence synthesis methodology. They can be used to map the evidence related to any topic to allow examination of practice, methods, policy and where (and how) future research could be undertaken. As such, they are a useful form of evidence synthesis for pharmacy clinicians, researchers and policymakers to review a broad range of evidence sources. COMMENT: This commentary presents the most comprehensive and up to date methodology for scoping reviews published by Joanna Briggs Institute (JBI). This approach builds upon two older approaches by Arksey and O'Malley, and Levac. To assist reviewers working in the field of pharmacy with planning and conducting scoping reviews, this paper describes how to undertake scoping reviews from inception to publication with specific examples related to pharmacy topics. WHAT IS NEW AND CONCLUSION: The JBI scoping review methodology is a valuable evidence synthesis approach to the field of pharmacy and therapeutics. This approach can assist pharmacy clinicians, researchers and policymakers to gain an understanding of the extant literature, to identify gaps, to explore concepts, characteristics and to examine current practice.

Systematic Reviews and Quality Appraisal of In Vitro Cancer Studies: Investigation of Current Practice.

BACKGROUND/AIM: To assess the quantity and quality of systematic reviews of in vitro cancer studies. MATERIALS AND METHODS: PubMed, MEDLINE, Embase, Web of Knowledge and PROSPERO databases were searched. Articles described as systematic reviews of in vitro studies, focused on or relevant to cancer and published in English were selected and appraised using an adapted version of AMSTAR 2 'critical domains'. RESULTS: From 4,021 records, 41 reviews described as systematic and cancer-related were identified. Publication dates indicate increasing frequency of systematic review conduct. Mean number of databases searched was three (range=1-8). Thirty-six reviews (88%) reported search methods, 35 (85%) specified inclusion criteria, 26 (63%) reported study selection methods, and 21 (51%) used reporting guidelines. Only 13 reviews (32%) involved formal quality assessment. CONCLUSION: Detailed investigation of reviews of cancer-relevant in vitro studies indicates need for further development and use of robust search strategies, appropriate quality assessment tools, and researchers with relevant skills.

How to Interpret and Use a Clinical Practice Guideline or Recommendation: Users' Guides to the Medical Literature.

IMPORTANCE: Clinicians may rely on recommendations from clinical practice guidelines for management of patients. OBSERVATIONS: A clinical practice guideline is a published statement that includes recommendations that are intended to optimize patient care. In the guideline development process, a panel of experts formulates recommendation questions that guide the retrieval of evidence that is used to inform the recommendations. Typically, methods of guideline development, a summary of the supporting evidence, and a justification of the panel's decisions accompany the recommendations. To use such guidelines optimally, clinicians must understand the implications of the recommendations, assess the trustworthiness of the development process, and evaluate the extent to which the recommendations are applicable to patients in their practice settings. Helpful recommendations are clear and actionable, and explicitly specify whether they are strong or weak, are appropriate for all patients, or depend on individual patients' circumstances and values. Rigorous guidelines and recommendations are informed by appropriately conducted, up-to-date systematic reviews that consider outcomes important to patients. Because judgments are involved in the interpretation of the evidence and the process of moving from evidence to recommendations, useful guidelines consider all relevant factors that have a bearing in a clinical decision and are not influenced by conflicts of interest. CONCLUSIONS AND RELEVANCE: In considering a guideline's recommendations, clinicians must decide whether there are important differences between the factors the guideline panel has considered in making recommendations and their own practice setting.

Evaluating agreement between bodies of evidence from randomised controlled trials and cohort studies in nutrition research: meta-epidemiological study.

OBJECTIVE: To evaluate the agreement between diet-disease effect estimates of bodies of evidence from randomised controlled trials and those from cohort studies in nutrition research, and to investigate potential factors for disagreement. DESIGN: Meta-epidemiological study. DATA SOURCES: Cochrane Database of Systematic Reviews, and Medline. REVIEW METHODS: Population, intervention or exposure, comparator, outcome (PI/ECO) elements from a body of evidence from cohort studies (BoE(CS)) were matched with corresponding elements of a body of evidence from randomised controlled trials (BoE(RCT)). Pooled ratio of risk ratios or difference of mean differences across all diet-disease outcome pairs were calculated. Subgroup analyses were conducted to explore factors for disagreement. Heterogeneity was assessed through I2 and τ2. Prediction intervals were calculated to assess the range of possible values for the difference in the results between evidence from randomised controlled trials and evidence from cohort studies in future comparisons. RESULTS: 97 diet-disease outcome pairs (that is, matched BoE(RCT) and BoE(CS)) were identified overall. For binary outcomes, the pooled ratio of risk ratios comparing estimates from BoE(RCT) with BoE(CS) was 1.09 (95% confidence interval 1.04 to 1.14; I2=68%; τ2=0.021; 95% prediction interval 0.81 to 1.46). The prediction interval indicated that the difference could be much more substantial, in either direction. We further explored heterogeneity and found that PI/ECO dissimilarities, especially for the comparisons of dietary supplements in randomised controlled trials and nutrient status in cohort studies, explained most of the differences. When the type of intake or exposure between both types of evidence was identical, the estimates were similar. For continuous outcomes, small differences were observed between randomised controlled trials and cohort studies. CONCLUSION: On average, the difference in pooled results between estimates from BoE(RCT) and BoE(CS) was small. But wide prediction intervals and some substantial statistical heterogeneity in cohort studies indicate that important differences or potential bias in individual comparisons or studies cannot be excluded. Observed differences were mainly driven by dissimilarities in population, intervention or exposure, comparator, and outcome. These findings could help researchers further understand the integration of such evidence into prospective nutrition evidence syntheses and improve evidence based dietary guidelines.

Methodological quality of systematic reviews comprising clinical practice guidelines for cardiovascular risk assessment and management for noncardiac surgery.

BACKGROUND: Cardiac assessment in noncardiac surgery clinical practice guidelines should be supported by the highest-quality evidence such as that offered by systematic reviews. Currently, the methodological and reporting quality of these studies remains unknown. METHODS: We used PubMed to search for all clinical practice guidelines related to perioperative cardiovascular patients undergoing noncardiac surgery from 2010 to 2021. The included clinical practice guidelines were analysed for all systematic reviews and meta-analyses. The primary objective of this study was to determine reporting and methodological quality using the PRISMA (Preferred Reporting Instrument for Systematic Reviews and Meta-Analyses) and AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews-2) instruments. Our secondary objective was to compare systematic reviews conducted by the Cochrane Collaboration with non-Cochrane studies. RESULTS: Three clinical practice guidelines were included in our study. Within these, 78 systematic reviews were included. PRISMA completion ranged from 34.8% to 100.0% with a mean of 76.9%. AMSTAR-2 completion ranged from 15.6% to 96.9% with a mean of 58.0%. Fifty-four systematic reviews underpinned a clinical practice guidelines recommendation, of which 25 were rated 'critically low' by AMSTAR-2 appraisal. Cochrane systematic reviews typically performed better than non-Cochrane studies, but were a minority of the included studies (10/78). CONCLUSION: We found deficiencies in several key areas regarding the methodological and reporting qualities of systematic reviews included in cardiac assessment in noncardiac surgery clinical practice guidelines. As these clinical practice guidelines are instrumental to clinical decision-making and patient care in cardiac assessment in noncardiac surgery, we advocate for improved reporting quality among systematic reviews cited as supportive evidence for these recommendations.

The quality of systematic reviews and other synthesis in the time of COVID-19.

COVID-19 research has been produced at an unprecedented rate and managing what is currently known is in part being accomplished through synthesis research. Here we evaluated how the need to rapidly produce syntheses has impacted the quality of the synthesis research. Thus, we sought to identify, evaluate and map the synthesis research on COVID-19 published up to 10 July 2020. A COVID-19 literature database was created using pre-specified COVID-19 search algorithms carried out in eight databases. We identified 863 citations considered to be synthesis research for evaluation in this project. Four-hundred and thirty-nine reviews were fully assessed with A MeaSurement Tool to Assess systematic Reviews (AMSTAR-2) and rated as very low-quality (n = 145), low-quality (n = 80), medium-quality (n = 208) and high-quality (n = 151). The quality of these reviews fell short of what is expected for synthesis research with key domains being left out of the typical methodology. The increase in risk of bias due to non-adherence to systematic review methodology is unknown and prevents the reader from assessing the validity of the review. The responsibility to assure the quality is held by both producers and publishers of synthesis research and our findings indicate there is a need to equip readers with the expertise to evaluate the review conduct before using it for decision-making purposes.

AGREE II appraisals of clinical practice guidelines in rehabilitation showed poor reporting and moderate variability in quality ratings when users apply different cuff-offs: a methodological study.

OBJECTIVES: To analyze the reporting characteristics of Appraisal of Guidelines Research and Evaluation (AGREE) II appraisals in rehabilitation and explore how much quality ratings of Clinical Practice Guidelines (CPGs) vary applying different cut-offs. STUDY DESIGN AND SETTING: We conducted a methodological study re-analyzing data of an overview of AGREE II CPG appraisals in rehabilitation. Reporting characteristics of appraisals and methods used for quality rating were abstracted. We applied the most frequent cut-offs retrieved on all CPG sample to explore changes in quality ratings (i.e., high/low). RESULTS: We included 40 appraisals (n = 544 CPGs).The AGREE II overall assessment 1 (overall CPG quality) was reported in 26 appraisals (65%) and the overall assessment 2 (recommendation for use) in 17 (42.5%). Twenty-five appraisals (62.5%) reported the use of cut-offs based on domains and/or overall assessments. Application of the most reported cut-offs led to variability in quality ratings in 26% of the CPGs, of which 92% CPGs shifted their rating from low to high-quality and 8% shifted from high to low-quality. CONCLUSION: Rehabilitation stakeholders should take care to select the highest quality CPG in view of the poor reporting of AGREE II overall assessment 1 and 2 and moderate variability of quality ratings.

Cochrane's risk of bias tool for non-randomized studies (ROBINS-I) is frequently misapplied: A methodological systematic review.

OBJECTIVES: We aimed to review how 'Risk of Bias In Non-randomized Studies-of Interventions' (ROBINS-I), a Cochrane risk of bias assessment tool, has been used in recent systematic reviews. STUDY DESIGN AND SETTING: Database and citation searches were conducted in March 2020 to identify recently published reviews using ROBINS-I. Reported ROBINS-I assessments and data on how ROBINS-I was used were extracted from each review. Methodological quality of reviews was assessed using AMSTAR 2 ('A MeaSurement Tool to Assess systematic Reviews'). RESULTS: Of 181 hits, 124 reviews were included. Risk of bias was serious/critical in 54% of assessments on average, most commonly due to confounding. Quality of reviews was mostly low, and modifications and incorrect use of ROBINS-I were common, with 20% reviews modifying the rating scale, 20% understating overall risk of bias, and 19% including critical-risk of bias studies in evidence synthesis. Poorly conducted reviews were more likely to report low/moderate risk of bias (predicted probability 57% [95% CI: 47-67] in critically low-quality reviews, 31% [19-46] in high/moderate-quality reviews). CONCLUSION: Low-quality reviews frequently apply ROBINS-I incorrectly, and may thus inappropriately include or give too much weight to uncertain evidence. Readers should be aware that such problems can lead to incorrect conclusions in reviews.